Internal lighting for endoscopic organ visualization

ABSTRACT

Systems and methods are provided rendering a three-dimensional volume. Scan data representing an anatomical object of a patient is acquired. A boundary of the object is identified in the scan data. A first lightmap is positioned inside the boundary of the object. A second lightmap is positioned outside the boundary of the object. The three-dimensional volume of the object is rendered from the scan data with lighting based on the first lightmap and second lightmap.

BACKGROUND

The present embodiments relate to medical imaging and visual lightingtechniques.

Virtual endoscopy uses medical imaging, such as computed tomography (CT)or magnetic resonance (MR) scanning, combined with computer imaging toprovide a non-invasive view of internal objects. Examples include scansof the abdominal area, the heart, the head, the lungs, rotationalangiography of blood vessels in various body parts, and others. Based onthe resulting volumetric data, the organs of interest are visualized andinspected from interior, e.g. “endo,” viewpoints.

Using virtual endoscopy, a user may conduct virtual examinations ofinternal regions of a patient, simulating the way an actual endoscopyviews the internal regions. Virtual endoscopy overcomes a traditionalendoscopy's disadvantage that requires inserting a scope into apatient's body. As opposed to traditional endoscopy, virtual endoscopymay be a completely non-contact inspection method. Virtual endoscopy hasmany uses including teaching, diagnosis, intervention planning, andinteroperative navigation among other uses. Due to the non-invasivenature, virtual endoscopy may be less risky and less expensive to apatient and/or hospital.

As a simulated procedure, virtual endoscopy functions best when theimage is presented in a manner that is easy to view and analyze. Thescan data collected by the medical imaging devices may provide structureand some texture, but lacks lighting. Artificial lighting may be addedusing computer rendering techniques. Prior implementations for virtualendoscopic rendering have used a synthetic point light at the camera ora directional light as a light source. The use of a single point lightsource generates shadows and dark regions in the image that prevent aproper analysis. Using multiple point sources may dramatically increasethe computational requirement, preventing a system from providing realtime imaging. Using an average or ambient illumination for the entirevolume results in an unrealistic view.

SUMMARY

By way of introduction, the preferred embodiments described belowinclude methods, computer readable media and systems for generatingphotorealistic views of an internal object using lightmaps. Multiplelightmaps may be placed and used together with optional syntheticlighting and path tracing-based rendering in order to enablephotorealistic, in-context internal views. A first lightmap ispositioned inside the object to illuminate the internal object and innerobject boundary. A second lightmap is positioned outside an object witha certain distance from an outer object boundary to provide the mainillumination on external objects. A translucent window on a region ofthe object may provide a depiction of both the inside of the object andany external objects as context. A photorealistic view of the internalobject and external object may be rendered using path tracing and thefirst and second lightmaps.

In a first aspect, a method is provided for rendering athree-dimensional volume. Scan data representing an anatomical object ofa patient is acquired. A boundary of the object is identified in thescan data. A first lightmap is positioned inside the boundary of theobject. A second lightmap is positioned outside the boundary of theobject. The three-dimensional volume of the object is rendered from thescan data with lighting based on the first lightmap and second lightmap.

In a second aspect, a method is provided for generating a photorealisticimage of an organ. Scan data of the organ is acquired. The scan data isrendered to an image with illumination based on a first lightmappositioned inside the organ and a second lightmap positioned outside theorgan.

In a third aspect, a system is provided for rendering athree-dimensional volume. The system includes a memory, a graphicsprocessing unit, and a processor. The memory is configured for storingdata representing an object in three dimensions. The graphics processingunit is configured to render illumination from a first lightmappositioned inside an object and a second lightmap positioned outside theobject. The processor is configured to render an image of the objectincluding the illumination.

The present invention is defined by the following claims, and nothing inthis section should be taken as a limitation on those claims. Furtheraspects and advantages of the invention are discussed below inconjunction with the preferred embodiments and may be later claimedindependently or in combination.

BRIEF DESCRIPTION OF THE DRAWINGS

The components and the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of the invention.Moreover, in the figures, like reference numerals designatecorresponding parts throughout the different views.

FIG. 1 is a flow chart of one embodiment of a method for providingphotorealistic views of an internal object in a medical system;

FIG. 2 illustrates a cross section view of an example internal object;

FIG. 3 illustrates a cross section view of the example internal objectof FIG. 2 with an internal lightmap;

FIG. 4 illustrates a cross section view of the example internal objectof FIG. 2 with an external lightmap; and

FIG. 5 is a block diagram of one embodiment of a medical system forproviding photorealistic views of an internal object in a medicalsystem.

DETAILED DESCRIPTION

Two lightmaps are used for path tracing-based volume rendering invirtual endoscopy. A boundary of an object of a patient is identified inscan data. A first lightmap is positioned inside the object. A secondlightmap is positioned outside the object. The first lightmap providesillumination values for the interior of the object. The second lightmapprovides illumination values for organs outside the object. Athree-dimensional volume is rendered using a path tracing renderer withillumination rendered as a function of the first and second lightmaps.

Providing medical imaging with realistic lighting may be challenging.Medical volumes, from CT or MRI medical scan data, may be generated toprovide a non-invasive view of an internal organ. Virtual endoscopy orcomputed endoscopy for example, are methods of imaging to assistdiagnosis. The virtual endoscopy uses computer processing of the medicalvolume to provide simulated visualizations of patient specific organssimilar or equivalent to those produced by standard endoscopicprocedures.

For generating a medical image, surface and volume rendering use avariety of shading techniques and/or application of generic texturemappings related to specific surfaces. In order to visualize thevolumes, a synthetic light source may be used to illuminate an internalobject. A synthetic point light at a position of a camera viewpoint, ora directional light as a light source may be used to approximate theinternal lighting of an actual endoscopy. Rendered lighting may involvetwo different types of illumination, direct and global illumination.Direct illumination may be the result of the light that hits a surfacedirectly from a light source. Global illumination models how light isbounced off of the surface onto other surfaces. Multiple techniques maybe used for rendering both direct and global illumination.

Ray tracing and path tracing are two different techniques for renderingthe medical volume to a two-dimensional display image. Ray tracing maygenerate an image by tracing the path of light or vision through pixelsin an image. Ray tracing tracks the encounters of the ray of light withthe voxels of the medical volume. For example, the ray of light maycollide with a virtual object at an angle. The ray of light may thenreflect off into another virtual object. Each reflection or absorptionmay provide an illumination value that is captured and used by the raytracing to generate the image.

In an example, light rays are traced from a virtual camera into avolume. Where the light rays intersect with objects, the rays arescattered and generate multiple rays to each light in the volume. Pixelvalues for lighting are calculated based on the material properties ofthe object with the amount of light that pixel is receiving from all thelights in the volume. Ray tracing is limited in that ray tracing onlycalculates direct lighting. For environmental or global lighting, raytracing may sum all the direct lighting in the volume and apply thevalue across all the pixels in the volume (in addition to directlighting). The result may appear unrealistic as the environmentallighting is uniformly distributed.

As opposed to ray tracing, path tracing computes global illuminationbased on actual light bounces. Path tracing is similar to ray tracing inthat rays are cast from a virtual camera and traced through a simulatedscene. Path tracing uses random sampling to incrementally compute afinal image. Light photons with absorption and scattering based on themedical volume are modeled. Due to the random sampling, many photonsalong each viewing direction are modeled, each with random scatteringand absorption forming a path through any number of scatter events. Therandom sampling process, e.g. a Monte Carlo algorithm for the path ofeach photon, allows for path tracing to render complex phenomena thatare not computed in regular ray tracing.

In path tracing, rays are distributed randomly within each pixel incamera space, and, at each intersection with an object in the volume, anew reflection ray, pointing in a random direction, is generated. Aftera number of bounces, each ray exits the volume or is absorbed. When aray (e.g., photon) has finished bouncing about in the volume, a value iscalculated based on the objects the ray bounced against. The value isadded to the average for the source pixel. Path tracing functions wellfor direct lighting, but may have issues rendering indirect lighting.Path tracing may use a radiosity method which divides a surface of anobject into a large number of patches and computes how much each patchcontributes to the illumination. Radiosity may be very inefficient, andunable to provide real time photorealistic images.

The samples in a path traced image are distributed over all pixels. Thevalue of each pixel is the average of all the sample values computed forthat pixel. The random components in path tracing cause the renderedimage to appear noisy. The noise decreases over time as more samples arecalculated. Generating a photorealistic image from path tracing may usea large computational requirement. The number of samples or bounces maybe limited to increase efficiency, but may result in a less realisticimage. Additionally, similarly to ray tracing, the environmentallighting in path tracing may result in a non-realistic image. With onlya synthetic point source, the lighting may appear artificial to a user.For procedures and diagnosis that rely on a user's ability to detect andanalyze the volume, non-realistic lighting may disadvantage the user.

Furthermore, use of point source lighting regardless of renderingtechnique may not provide context for a volume. Light from the pointsource may be blocked by walls of the internal object and/or may createshadows or dark areas in the volume. Simulated light rays from the pointsource may hit an object that blocks the rays, absorbs the rays, orreflects the rays. Areas outside the object or organ may not beilluminated, leaving a user without spatial contextual information.

With only a point light source, a virtual endoscopic image may bedifficult to analyze. Multiple point light sources may be used, but theincrease in computational requirements may prevent an image from beingrendered in real time in response to user actions. A global illuminationlevel may be increased across the entire image. The increase, however,may lead to unrealistic views of the object and exterior organs.

Methods and systems are provided that provide for realisticenvironmental lighting in medical imaging with path tracing for virtualendoscopy. Medical images are received. A boundary of an object isidentified from the medical images. A first lightmap is positioned at afirst distance inside the boundary. A second lightmap is positioned at asecond distance outside the boundary. A medical volume is rendered bypath tracing using the first lightmap and second lightmap. A translucentwindow may be rendered on the boundary.

The use of two lightmaps provides for an efficient realistic lightingthat provides external context for a user. The use and positioning ofthe two lightmaps provide for environmental mapping and reflectionmapping that may be used by a path-tracing renderer. The two lightmapsstore a precomputed texture that may take full advantage of complexlighting textures that control the appearance of an object.

FIG. 1 depicts a diagram of one embodiment of a method for providingphotorealistic images in a medical system. The method relates to the useand positioning of two lightmaps to provide lighting textures to animage rendered using a path tracing algorithm.

The medical system of FIG. 5 or other medical system implements theacts. The system may be a medical imaging system, a hospitalworkstation, a medical server, or other secure medical data processingsystem. The medical system may or may not include a memory or database,such as patient medical record database and/or picture archiving andcommunications system (PACS).

The acts of FIG. 1 are performed in the order shown (numerical or top tobottom) or other orders. For example, acts A130 and A140 occursimultaneously or in any order. Additional, different, or fewer acts maybe provided. Example additional acts include addition of a point sourcelight or adjustment of a transparent section of the organ.

Images rendered using global illumination algorithms may appear morephotorealistic than those using only direct illumination algorithms.However, such images are computationally more expensive and consequentlymuch slower to generate. One common approach is to compute the globalillumination of a scene and store that information with the geometry ofthe volume. The stored data may then be used to generate images fromdifferent viewpoints for generating walkthroughs of a volume withouthaving to go through expensive lighting calculations repeatedly.

At act A110, the system acquires scan data representing anatomicalobjects of a patient. The scan data may be provided from a memory, amedical scanner, sensors, and/or other source. The data may be formattedas voxels. Each voxel may be represented by 3D location (e.g., x, y, z)and an intensity, scalar, or other information. In one embodiment, thescan data represents a patient. In the examples below, medical imagingor scanner data is used. In other embodiments, other types of data areused. A medical scanner may provide the data, such as a medical datasetrepresenting a 3D region of the patient. Any type of medical data isobtained, such as computed tomography, magnetic resonance, positronemission tomography, single photon emission computed tomography,ultrasound, or another scan modality. Scan data representing a 3D volumeis loaded as a medical dataset. The scan data may be from multipletwo-dimensional scans or may be formatted from a 3D scan. The scan datamay represent one or more objects, for example, the internal organs of apatient. The scan data may include an object such as a lumen, e.g. ahollow object that may be the focus of a virtual endoscopic procedure.Example lumens include vessels, organs of the digestive system,reproductive organs, the heart, ear canal, respiratory system, or otherorgans, tracts, or parts of the body with a lumen.

At act A120, the system identifies a boundary of an internal object ofthe patient in the scan data. The scan data may include one or morelumens or objects inside a patient. A lumen, for example, may representa cavity or internal area of a tubular or hollow object. The objects mayinclude other lumens, organs, or objects inside a patient. The boundaryof the object may approximate the walls of the object. The boundary maybe defined by a distance (or in the case of a hollow tube, a radius)from a centrally located point from the interior walls of the object.The central point may track or represent a location of a virtual cameraplaced inside the object. The central point may shift over time as thevirtual camera travels through the object or may be in one positionwithout shifting.

The boundary may be used to position the two lightmaps. An internallightmap outside the boundary may be prevented from providing internalillumination or reflectance. An external lightmap inside the boundary,similarly would not be able to provide illumination or reflection forexternal objects.

The boundary may be identified using a segmentation algorithm.Thresholding, random walker, or other segmentation approaches may beused.

The segmented scan data may include a wireframe model or shape model ofthe interior of the patient without textures. The segmentation algorithmmay assign a label to each pixel or voxel in the scan data. Similarlabeled pixels or voxels may indicate similar objects that may providefor an edge or object boundary to be identified in the scan data. Theboundary may be defined by the walls of the object, e.g. defined by theinterior wall and exterior wall. Alternatively, the boundary may be asingle line that approximates the walls of the object. In a 2D segmentof the volume, the boundary may appear circular or ellipsoidal. For the3D volume, the boundary may be cylindrically, ovoidal, or sphericallyshaped. Non-regular shapes may be used. The boundary may be stored in amemory as a collection of pixels or coordinate values. A dilationalgorithm and/or filtering may be applied to enlarge and smooth thesegmentation results.

While an accurate boundary may be desirable, the boundary may not needto be accurate to a certain degree. Automatic segmentation or a manualsegmentation may be used to identify the boundary. A rough approximationof the boundary, for example, within 1, 5, or 10 voxels may be used. Anestimation of the center of a wall of the object may be used with anestimated thickness to generate an estimated boundary. Alternative edgedetection methods may be used. For example, an iso-surface, e.g. asurface that represents points of a constant value within a volume ofspace may also be used instead of object segmentation.

FIG. 2 depicts an example 2D cross section view of a 3D volume includingan identified boundary 215. The figure includes an object 210 and anexternal object 220 outside the object 210. FIG. 2 further includes avirtual camera 230 placed at a camera position and a field of view 250of the virtual camera 230. The field of view 240 includes a section ofthe object wall 215. The walls 215 of the object 210 have been depictedas a single line, however, in practice the object walls 215 may vary inwidth along the object 215. The walls 215 in FIG. 2 may be identified asthe boundary in act A120.

At act A130, a first lightmap, or internal lightmap, is positionedinside the boundary 215. A lightmap may be a type of texture map thatmay be overlaid on an object to provide lighting detail or lightingvalues for rendering an image.

The positioning of the lightmaps is a function of distances relative tothe object boundary 215. Improperly placed lightmaps within objects mayintroduce undesirable dark unlit areas and artifacts. The lightmap maybe placed within the boundary 215 so that a selected center point of thelightmap is a minimum distance from the boundary 215. The selectedposition may be a position of a virtual camera. The object walls (and assuch, boundary 215) may not be equidistant from the selected position.The first distance is the minimum distance possible so that, theentirely of the internal lightmap is inside the object and does notintersect with the boundary 215. In an example of a spherical lightmap,the entire sphere with a radius of the minimum distance fits within theobject. In alternative embodiments, the lightmap intersects with theboundary 215.

Lightmaps may include elements that include lighting parameters referredto as lumels. The size of the lumels relates to the amount of detail inthe map. Smaller lumels yield a higher resolution lightmap, providingfiner lighting detail with a drawback of reduced performance andincreased memory usage. For example, a lightmap scale of 2 lumels perunit may give a lower quality than a scale of 8 lumels per unit. Theresolution of the lightmap may also be limited by the amount of diskstorage space, bandwidth/download time, or texture memory available tothe application.

The resolution of the lightmap may be dependent on the first distance. Alarger lightmap, for example, may contain more lumels and as such, moreinformation. For a spherical lightmap, a smaller radius leads to asmaller surface area for the lightmap, and less area to store andproject lighting information. The surface area corresponds to theresolution that is the area, in pixels, available for storing one ormore surface's lighting values. A spherical lightmap that has a selectedpoint at the minimum distance from the boundary 215 may result in thelargest possible amount of data stored in the lightmap withoutintersecting with the boundary 215.

The lightmaps may be generated in real time for each frame of the volumeimage. The lightmaps may be pre-computed. The lighting parameters storedin a lightmap may be derived from the type of tissue or object. Thelighting parameters may be defined by a user or set at a default level.The lighting parameters may include resolution, irradiance, reflectance,intensity, translucency, absorbance, and others. An interior wall of acolon, for example, may have a range of reflectance values. A section ofan object may have a predefined intensity. An interior wall of a stomachmay include different values for irradiance or reflectance. Eachmaterial may have different values for different scenarios. The lightingparameters for the material may be globally used or may be patientspecific. The lighting parameters may be derived from perviousendoscopic procedures and images captures with a physical camera andlighting system. The lighting parameters may be defined by a user.

The internal lightmap may be a reflectance lightmap. A reflectancelightmap includes reflectance values for the reflective properties ofthe surfaces in the volume. The reflective properties (amount, directionand colour) of surfaces may be modeled using a bidirectional reflectancedistribution function (BRDF). BRDF is a function of four variables thatdefines how light is reflected at an opaque surface. The equivalent fortransmitted light (light that goes through the object) is abidirectional scattering distribution function (BSDF). The lightmaps mayuse either or both the BRDF or BSDF functions to control the lightingvalues of an object.

The lightmap may be a spherical, cubical, rectangular, cylindrical,shaped to confirm with the lumen, or other shaped map. Spherical mappingprovides illumination as though the illumination was seen in thereflection of a reflective sphere through an orthographic camera. Cubemapped reflection is done by determining the vector that the object isviewed at. The camera ray is reflected about the surface of where thecamera vector intersects the object. The reflected ray is then passed tothe cube map to retrieve the lumel that provides the radiance value usedin the lighting calculation. The rendered radiance value creates theeffect that the object is reflective.

FIG. 3 depicts the positioning of the internal lightmap 310 inside theobject 210. In FIG. 3, the internal lightmap 310 is positionedcompletely inside the focus object 210 illuminating the internaltissues. The internal lightmap 310 may be positioned centered at aposition of the virtual camera 230, with the radius scaled with theminimum distance 330 from the virtual camera 230 to the object boundary215. If the radius is scaled to exactly the minimum distance, a singlepoint on the lightmap may touch the object boundary 215. As depicted inFIG. 3, the distance from the position of the virtual camera 230 to theobject boundary 215 varies. For example, the distance 320 is larger thanthe distance 330. As the distance 330 is the minimum distance, thatdistance is chosen for positioning the lightmap. Alternatively, if thecamera position was shifted to the left in FIG. 3, the distance 320 maybe shorter and as such, the minimum distance.

The internal lightmap 310 may be positioned at a selected point insidethe object other than the camera position. The internal lightmap 310 maybe positioned so that the radius is smaller than the minimum distancefrom the selected position to the object boundary 215. The selectedpoint may be, for example, a center point in the object. A center pointmay reside on a centerline that may be identified by calculating aperpendicular distance from each point on the boundary 215 andidentifying the center.

FIG. 3 depicts a single image frame. For a procedure that requiresmultiple image frames, the camera and/or object may move or be altered.The movement of the camera or object may result in a change in theminimum distance from the position of the internal lightmap 310. Theminimum distance may thus be adjusted for each frame. In order tomaintain a high rendering framerate when the camera and/or object moves,an optimization technique to estimate the minimum distance may be usedto precompute a distance volume of an acceptable resolution containingthe minimum distance to the boundary 215 around the object. At eachcamera position, a lookup into the distance volume provides the minimumdistance value at render time. The internal lightmap may be positionednot at a camera position, but at a center point or other selected pointinside the object. A predefined, e.g. not interactive point, may providefor a high rendering framerate.

At act A140, a second lightmap, also referred to as an externallightmap, is positioned outside the boundary 215. The external lightmapmay be an illuminance lightmap, a reflectance lightmap, or other type oflightmap. The external lightmap may be used to illuminate thesurrounding organs beyond the focus object walls or provide lightingparameters of the exterior surfaces or tissues. The illumination outsidethe focus object walls may be used to support a translucent window oropaque organs in the rendered volume. The external lightmap may bepositioned centered on a point at the camera location. The radius, inthe case of spherical or cylindrical map, may be at least large enoughto contain the focus object and not intersect with the boundary 215 inthe viewing field. To achieve interactive computation of the maximumdistance, a precomputed maximum distance volume may be used.Alternatively, for the maximum distance, a depth image of the objectboundary 215 may be rendered with the current camera settings. Themaximum depth value in the depth image is used to estimate the maximumdistance of the object boundary 215 to the camera. The value may bespecified or adjusted by the application.

The external lightmap may be pre-computed using rendering techniquessuch as ambient occlusion, phong shading, or photon mapping.Alternatively, the external lightmap may be generated in real timeduring a procedure.

The external lightmap may be adjusted to cover a larger volume toinclude or exclude other objects. For example, the size of the externallightmap may be adjusted to cover only an adjacent external objectinstead of the entirety of the patient. The external lightmap may beadjusted to cover a specific region or cavity of a patient.

FIG. 4 depicts a positioning of the external lightmap 410. The externallightmap 410 is positioned outside the focus object 210, illuminatingthe external organs 220. The external lightmap 410 may be positionedcentered at the position of the virtual camera 230, with a radius scaledwith at least the maximum distance from the camera to the objectboundary 215. The maximum distance here may be represented by thedistance 420. The external lightmap 410, as depicted in FIG. 4, ispositioned to illuminate the entirely of the external object 220. FIG. 4further depicts a translucent (or transparent) window 430 of the objectwalls so that a user may view the exterior of the object when the fieldof view 250 includes the translucent window 430. The entire boundary 215may be partially translucent. FIG. 4 also depicts the internal lightmap310 that may be used to provide lighting for the interior of the focusobject 210.

In an embodiment, additional light sources may be used. One or morefeatures may be detected in the scan data. A user may desire to see thefeature with a separate light source. The light source may be placed atthe feature point, above the feature point, or at a back of the featurepoint based on the use cases. Optionally, a light source such as pointlight or directional light may be used in combination with the lightmapsby using local shading techniques (for example, Phong shading) when theray (for ray tracing) reaches a surface. The light source may bepositioned at the camera location to simulate a traditional endoscopicprocedure.

At act A150, an image is rendered from the scan data with lighting basedon the internal lightmap 310 and external lightmap 410. The image mayinclude a view from a virtual camera positioned inside the object. Theimage may include a view of externally objects outside the focus object,for example, by rendered a portion of the object transparent or notfully opaque. The scan data may provide a skeleton or wire mesh frame onwhich illumination textures are added by a rendering process based onthe lightmaps 310, 410. The textures may include shading, color, andlighting components. The three-dimensional image may be rendered to bephoto-realistic.

Path tracing may be used to render the scan data and textures. Usingpath tracing, light rays bounce around the volume, acquiring values thatthe path tracing algorithm uses to solve the rendering equation. A raymay collide with a surface of an object with a high reflectivity (energyof ray after hitting the surface), with some surface graininess(reflection/refraction) and so on. The attributes, such as thereflectivity of the tissues or objects may be defined by the lightmaps.As a ray continues to bounce around, each ray absorbs, reflects orsplits into multiple new rays depending on the properties e.g. tissuethat the ray interacts with. The new rays also bounce around, performingthe same function. After a number of bounces the rays hit a lightsource, providing a final value, the initial amount of energy. Thelighting effects is rendered based on an algorithm that solves anequation including the values.

A Monte Carlo algorithm may be used to solve the equation. A Monte Carloalgorithm for rendering lighting is a statistical method based on anestimation of how much light is redirected to a point by other objectsin the volume by casting rays from the point in random directions abovethe surface and evaluating the values of the objects the rays intersect.The contribution of each one of the rays is then summed up and theresulting sum is divided by the total number of rays.

Volumetric path tracing-based rendering may be used to render the scandata. Volumetric path tracing provides path tracing with the effects oflight scattering. As in the path tracing method, a ray is tracedbackwards from the eye on until the ray reaches the light source. Involumetric path tracing, scatter events may occur during the tracing.When a light ray hits a surface, an amount of the ray may get scatteredinto the media. Volumetric path tracing samples a distance from thetransmittance along a ray. If the distance is less than the distance ofthe nearest surface intersection along the ray, a scatter occurred inthe media and the path is evaluated from the scatter point rather thanthe point on the surface.

In virtual endoscopic, external organs may be visualized to providecontextual information. For example, in virtual colonoscopy, an imagemay be rendered with fisheye lens of 180-degree field of view (FOV), anda user may want to see through the colon wall that is within 45 degreeFOV. Within the 45 degree FOV, the colon wall may be rendered assemi-transparent allowing the user to view the external organs outsidethe colon and in front of the camera. The portion of the wall that maybe rendered semi-transparent may be referred to as a translucent window.The translucent window may benefit other views where location context isimportant. For example, for a view of the heart, a camera is positionedinside the heart to view the valves. Simultaneously and in the sameimage, the coronary surrounding the heart surface that is in front ofthe camera may be displayed.

The translucent window 430 may be expanded to cover an entirety of acolon wall. The level of transparency of the translucent window may beadjusted by a user. A user, for example, may set the object wall to behighly transparent to determine the context of the region in which thevirtual camera is placed. After the user has determined the context, theuser may shift the transparency to a lower level in order to visualizethe object wall. The translucent window may be toggled on and off oradjusted by a user. The translucent window may be adjusted automaticallyto provide a view to the user.

The translucent window 430 and the view of the exterior organs orobjects may be rendered in a photorealistic way by using the externallightmap 410. Without the external lightmap 410, the exterior wouldeither not be illuminated or full of shadow effects or dark areas due tothe blocking nature of an object's walls. The illumination provided bythe interior lightmap may be separated from the external lightmap 410 toprovide accurate lighting without confusing shadows. Alternatively, theillumination of a point source inside the object may be combined withthe values from the exterior lightmap. In volumetric path tracing, at asampling position on the rays, an algorithm checks if the ray isscattered according to the scattering probability value at the samplingposition. When using a semi-translucent window or wall, if the samplingposition is inside the semi-transparent window 430 of the image, thescattering probability may be modulated by a user controlled value. Thehigher the user controlled value, the more transparent is thetranslucent window.

The exterior of the object may be lighted by either illumination fromthe exterior lightmap, point light sources, or light rays from theinterior of the object. The translucent (or transparent) window mayprovide scatter or absorption values for any rays that reflect or passthrough the translucent window. Alternatively, the interior and exteriorareas may be separately illuminated based on the two respectivelightmaps. The internal lightmap may provide illumination for theinterior. The external lightmap may provide illumination for theexterior. The lightmaps may be generated or adjusted individually ortogether to achieve a preferred lighting scheme.

The rendering of the scan data using the illumination and or reflectancevalues in the lightmaps results in a photorealistic image. A sequence ofimages may be provided as the image is built or rendered. Alternatively,for a given set of values of rendering parameters, a single image isoutput. The rendering parameters are a default set, set by the user,determined by a processor, or combinations thereof. The renderingparameters may include data consistency parameters. Data consistencyparameters include windowing, scaling, level compression, datanormalization, or others. The rendering parameters may include viewingdesign parameters. Viewing design parameters include type of camera,position of the camera, orientation of the camera, intrinsic parametersfor viewing, or others. One or more use-case specific parameters may beprovided. Use-case specific parameters are settings specific to a givenuse, such as a particular camera position for a given type of medicalreport or use of two cameras for stereoscopic viewing. Additionallighting parameters may be used to render the image. Lighting parametersmay include additional light sources such as point light sources placedautomatically or by a user. The additional light sources may includedifferent types or levels of lighting. Different contrast or coloringmay be used for different sources.

FIG. 5 shows a medical system for virtual endoscopy using internal andexternal lightmaps. The medical system includes a medical imaging system540, a processor 530, a memory 520, a GPU 510, and a display 550. Theprocessor 530, GPU 510, and the memory 520 are shown separate from themedical imaging system 540, such associated with being a computer orworkstation apart from the medical imaging system 540. In otherembodiments, the processor 530 and/or memory 520 are part of the medicalimaging system 540. In alternative embodiments, the medical system is aworkstation, computer, or server. For example, the medical imagingsystem 540 is not provided or is provided for acquiring datarepresenting a volume, and a separate database, server, workstation,and/or computer is provided for identifying and positioning lightmapsand rendering a volume with illumination based on the lightmaps.Additional, different, or fewer components may be used.

The system is used for rendering a volume of a patient from scan dataand two or more lightmaps. The system may be used to position lightmapsfor one or more image frames in a virtual endoscopic procedure.

The computing components, devices, or machines of the medical system,such as the medical imaging system 520 and/or the processor 530 areconfigured by hardware, software, and/or firmware to performcalculations or other acts. The computing components operateindependently or in conjunction with each other to perform any givenact, such as the acts of any of the methods described above. The act isperformed by one of the computer components, another of the computingcomponents, or a combination of the computing components. Othercomponents may be used or controlled by the computing components to scanor perform other functions.

The medical imaging system 540 is any now known or later developedmodality for scanning a patient. The medical imaging system 540 scansthe patient. For example, a C-arm x-ray system (e.g., DynaCT fromSiemens), CT like system, or CT system is used. Other modalities includeMR, x-ray, angiography, fluoroscopy, PET, SPECT, or ultrasound. Themedical imaging system 540 is configured to acquire the medical imagingdata representing the patient. The data is acquired by scanning thepatient using transmission by the scanner and/or by receiving signalsfrom the patient.

The memory 520 is a buffer, cache, RAM, removable media, hard drive,magnetic, optical, database, or other now known or later developedmemory. The memory 520 is a single device or group of two or moredevices. The memory 520 is within the system 540, part of a computerwith the processor 530, or is outside or remote from other components.

The memory 520 is configured to store medical scan data, other data,lightmap data, lightmap positioning, camera and point source lightingpositions, boundaries of internal objects in the medical scan dataand/or other information. Rendered volumes with illumination are storedin the memory 520. The memory 520 may store a pre-computed position andsize of each lightmap. The memory 520 may store pre-computed values forthe lightmap.

The memory 520 is additionally or alternatively a non-transitorycomputer readable storage medium with processing instructions. Thememory 520 stores data representing instructions executable by theprogrammed processor 530. The instructions for implementing theprocesses, methods and/or techniques discussed herein are provided oncomputer-readable storage media or memories, such as a cache, buffer,RAM, removable media, hard drive or other computer readable storagemedia. Computer readable storage media include various types of volatileand nonvolatile storage media. The functions, acts or tasks illustratedin the figures or described herein are executed in response to one ormore sets of instructions stored in or on computer readable storagemedia. The functions, acts or tasks are independent of the particulartype of instructions set, storage media, processor or processingstrategy and may be performed by software, hardware, integratedcircuits, firmware, micro code and the like, operating alone or incombination. Likewise, processing strategies may includemultiprocessing, multitasking, parallel processing and the like. In oneembodiment, the instructions are stored on a removable media device forreading by local or remote systems. In other embodiments, theinstructions are stored in a remote location for transfer through acomputer network or over telephone lines. In yet other embodiments, theinstructions are stored within a given computer, CPU, GPU, or system.

The processor 530 is a general processor, digital signal processor,three-dimensional data processor, graphics processing unit, applicationspecific integrated circuit, field programmable gate array, digitalcircuit, analog circuit, combinations thereof, or other now known orlater developed device for processing data. The processor 530 is asingle device, a plurality of devices, or a network. For more than onedevice, parallel or sequential division of processing may be used.Different devices making up the processor 530 may perform distinctfunctions, such as positioning a lightmap with as one device andrendering a volume with another device. In one embodiment, the processor530 is a control processor and/or graphics processing unit of themedical imaging system 540. In another embodiment, the processor 530 isone or more graphics cards. The processor 530 operates pursuant tostored instructions to perform various acts described herein.

The processor 530 is configured to receive scan data of an object,identify a boundary 215 and position lightmaps. The boundary 215 may beidentified in the scan data using segmentation. A first internallightmap may be positioned inside the object. A second external lightmapmay be positioned outside the object. The processor 530 or GPU 510renders a volume with illumination based on the lightmaps. Additionallight sources may be added to illuminated the object. The processor 530or GPU 510 may render a portion of the object translucent or transparentto provide a view from the interior of the object to the exterior of theobject.

The GPU 510 is a graphics chip, graphics card, multi-core processor orother device for parallel processing to perform volume rendering. TheGPU 510 is part of a computer, workstation, server, or mobile device.The GPU 510 is configured by software, hardware, and/or firmware toimplement volume rendering. Monte Carlo path tracing, volumetric pathtracing, or other technique for probabilistically or stochasticsimulation of scattering and/or absorption of photons is used to renderillumination for the volume. The processor 530 and GPU 510 may operatein sequence or parallel for generating and positioning the lightmaps andrendering the image volume. The GPU 510 is configured by an applicationprogramming interface to render an image from the 3D scan datarepresenting a patient. Using path tracing based rendering, aphotorealistic image is rendered. Path tracing is a method developed tosolve a rendering equation. Path tracing is a ray tracing technique, atechnique where rays are traced throughout a scene, hit some surface inthe scene and generates samples from the information of that surface.

The display 550 is a CRT, LCD, plasma, projector, printer, or otheroutput device for showing an image. The display 550 displays therendered volume. The display 550 may receive user input to adjust thedisplay image.

While the invention has been described above by reference to variousembodiments, it should be understood that many changes and modificationscan be made without departing from the scope of the invention. It istherefore intended that the foregoing detailed description be regardedas illustrative rather than limiting, and that it be understood that itis the following claims, including all equivalents, that are intended todefine the spirit and scope of this invention.

We claim:
 1. A method for rendering a three-dimensional volume, themethod comprising: acquiring scan data representing an object inside ofa patient; identifying in the scan data, a boundary of the object;providing a first lightmap inside the boundary of the object; providinga second lightmap outside the boundary of the object; and rendering thethree-dimensional volume of the object from the scan data with lightingbased on the first lightmap and second lightmap.
 2. The method of claim1, wherein rendering the three-dimensional volume comprises renderingwith volumetric path tracing.
 3. The method of claim 1, furthercomprising: rendering a portion of the boundary of the objecttransparent.
 4. The method of claim 3, wherein the transparent portionprovides a view of an external object from a camera position inside theobject.
 5. The method of claim 1, wherein the boundary is identifiedusing segmentation of the scan data.
 6. The method of claim 1, whereinthe boundary is a wall of the object.
 7. The method of claim 1, whereinthe first lightmap is spherical and centered at a camera position insidethe object, wherein the first lightmap has a radius smaller than aminimum distance from the camera position to the boundary.
 8. The methodof claim 1, wherein the second lightmap is spherical and centered at acamera position inside the object, wherein the second lightmap has aradius larger than a maximum distance from the camera position to theboundary.
 9. The method of claim 1, wherein the first lightmap comprisesa reflectance lightmap.
 10. The method of claim 1, wherein the secondlightmap comprises an illuminance lightmap.
 11. The method of claim 1,further comprising: providing a synthetic light source; wherein lightingfor the three-dimensional image is further rendered including thesynthetic light source.
 12. The method of claim 11, wherein thethree-dimensional image is further rendered using local shadingtechniques.
 13. A method for generating a photorealistic image of anorgan, the method comprising: acquiring scan data of the organ; andrendering the scan data to an image with illumination based on a firstlightmap positioned inside the organ and a second lightmap positionedoutside the organ.
 14. The method of claim 13, wherein the firstlightmap is spherical and centered at a camera position inside theorgan, wherein the first lightmap has a radius smaller than a minimumdistance from the camera position to a wall of the organ and wherein thesecond lightmap is spherical and centered at the camera position insidethe organ, wherein the second lightmap has a radius larger than amaximum distance from the camera position to the wall.
 15. The method ofclaim 13, wherein rendering to the image is volumetric path tracingrendering.
 16. The method of claim 13, further comprising: transparentlyrendering to the image with at least a portion of a wall of the organ.17. A system for rendering a three-dimensional volume, the systemcomprising: a memory configured to store data representing an object inthree dimensions; a graphics processing unit configured to renderillumination from a first lightmap positioned inside the object and asecond lightmap positioned outside the object; and a processorconfigured to render an image of the object including the illumination.18. The system of claim 17, wherein the first lightmap is spherical andcentered at a camera position inside the object, wherein the firstlightmap has a radius smaller than a minimum distance from the cameraposition to a wall of the object and wherein the second lightmap isspherical and centered at the camera position inside the object, whereinthe second lightmap has a radius larger than a maximum distance from thecamera position to the wall.
 19. The system of claim 17, wherein thegraphics processing unit is further configured to render illuminationbased on a point light source.
 20. The system of claim 17, wherein thegraphics processing unit is configured to render illumination usingvolumetric path tracing.